CSE 5243 - Introduction to Data Mining¶

Homework 4: Clustering¶

  • Student Name: Rita Brokhman
  • Student Email: brokhman.1@osu.edu
  • Student.#: 500646956 and brokhman.1

Instructions and Helpful Hints:

  • Consider putting all of your "discussion" text in markdown cells, not inline with code. That gives you more control over formatting. Markdown cheat sheet: https://www.markdownguide.org/cheat-sheet
  • Explain what you are doing, and why. Explain what you found out or learned.
  • Make sure you run your entire workbook before handing it in, so the output cells are populated.
  • Follow the Section structure as much as possible - put your content where it is requested, so we can find your answers.
  • If you have questions on expectations or need clarification or guidance, please ask. Post to Teams if it is a general question, so everyone benefits.

Introduction¶

Objectives:¶

In this lab, you will perform clustering on three datasets. Your will choose suitable clustering algorithms, evaluate them on the datasets, and compare their performance.

The objectives of this assignment are:

  1. Understand how to select and evaluate suitable off-the-shelf clustering algorithms based on the characteristics of a dataset and the outcomes you need.
  2. Understand how to tune and evaluate a clustering algorithm to achieve good performance.

Datasets:¶

  • The file small_Xydf.csv is a two-dimensional dataset with 200 records. It contains columns X0, X1, and y. The y column is the actual cluster number that was produced by the dataset generation algorithm. Do not use it for the clustering algorithm. It will be used to evaluate your clustering algorithm below.

  • The file large1_Xydf.csv is a two-dimensional dataset with 3000 records. It contains columns X0, X1, and y. The y column is the actual cluster number that was produced by the dataset generation algorithm. Do not use it for the clustering algorithm. It will be used to evaluate your clustering algorithm below.

  • The file large2_Xydf.csv is another two-dimensional dataset with 3000 records, and characteristics different from the “large1” dataset. It contains columns X0, X1, and y. The y column is the actual cluster number that was produced by the dataset generation algorithm. Do not use it for the clustering algorithm. It will be used to evaluate your clustering algorithm below.

Approach:¶

This homework makes use of the Clustering Algorithms offered by the SciKitLearn Library. Study the information at https://scikit-learn.org/stable/modules/clustering.html. Follow the guidance in the individual sections below.

Collaboration:¶

For this assignment, you should work as an individual. You may informally discuss ideas with classmates, to get advice on general Python usage, etc., but your work should be your own. Please make use of Microsoft Teams!

What you need to turn in:¶

  1. Code
  • For this homework, the code is the Jupyter Notebook. Use the provided Jupyter Notebook template, and fill in the appropriate information.
  • This homework requires you to use clustering algorithms in the SciKitLearn library. You also may use common Python libraries for I/O, data manipulation, data visualization, etc. (e.g., NumPy, Pandas, MatPlotLib,…). You may not use library operations that perform, in effect, the entire “core” computations for this homework. (e.g., If you happen to find a single function on the web that does essentially all of a major portion of the homework, you may not use it.) When in doubt, ask the grader or instructor.
  • The code must be written by you, and any significant code snips you found on the Internet and used to understand how to do your coding for the core functionality must be attributed. (You do not need to attribute basic functionality – matrix operations, IO, etc.)
  • The code must be commented sufficiently to allow a reader to understand the algorithm without reading the actual Python, step by step.
  • When in doubt, ask the grader or instructor.
  1. Written Report
  • For this homework, the report is the Jupyter Notebook. The report should be well-written. Please proof-read and remove spelling and grammar errors and typos. Please also include the HTML file, which helps the grading process.
  • The report should discuss your analysis and observations. Key points and findings must be written in a style suitable for consumption by non-experts. Present charts and graphs to support your observations. If you performed any data processing, cleaning, etc., please discuss it within the report.
  1. HTML File

Grading:¶

  1. Overall readability and organization of your report (10%) - Is it well organized and does the presentation flow in a logical manner; are there many grammar and spelling mistakes; do the charts/graphs relate to the text, etc.
  2. Evaluation of the K-means Clustering Algorithm on the Small Dataset (15%) – Is your configuration sound? Have you made an effort to tune the algorithm for good performance? Are your analyses and evaluations sound, and supported by suitable statistics and/or visualizations?
  3. Evaluation of the K-means Clustering Algorithm on the Large1 Dataset (15%) – Is your configuration sound? Have you made an effort to tune the algorithm for good performance? Are your analyses and evaluations sound, and supported by suitable statistics and/or visualizations?
  4. Evaluation of the K-means Clustering Algorithm on the Large2 Dataset (15%) – Is your configuration sound? Have you made an effort to tune the algorithm for good performance? Are your analyses and evaluations sound, and supported by suitable statistics and/or visualizations?
  5. Evaluation of the Second Clustering Algorithm on the Large2 Dataset (15%) – Is your choice of algorithm and your configuration sound? Have you made an effort to tune the algorithm for good performance? Are your analyses and evaluations sound, and supported by suitable statistics and/or visualizations?
  6. Evaluation of the Third Clustering Algorithm on the Large2 Dataset (15%) – Is your choice of algorithm and your configuration sound? Have you made an effort to tune the algorithm for good performance? Are your analyses and evaluations sound, and supported by suitable statistics and/or visualizations?
  7. Comparison of the Three Clustering Algorithms (10%) - Is the comparison sound? Did you choose a specific clustering algorithm as best and explain why?
  8. Conclusions (5%) – Did you document your overall insights?

How to turn in your work on Carmen:¶

Please follow these instructions exactly - it helps the grading process. If you have questions, please ask. Submit to Carmen any code that you used to process and analyze this data. You do not need to include the input data. All the related files (code and report) except for the data should be archived in a zip file (with no folder trees inside) and submitted via Carmen. The submitted file should be less than 5MB. Use this naming convention: HomeworkN_Surname_DotNumber.zip

References and Acknowledgements:¶

  1. https://scikit-learn.org/stable/modules/classes.html#module-sklearn.cluster
  2. https://scikit-learn.org/stable/modules/clustering.html
  3. https://docs.python.org/3/library/time.html
  4. https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.linear_sum_assignment.html
  5. https://gist.github.com/siolag161/dc6e42b64e1bde1f263b (using Hungarian Algorithm to match cluster labels - this is just an example)
  6. https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html (note that this is mean squared error)

Section: Overview¶

  • Insert a short description of the scope of this exercise, any supporting information, etc.

This homework focuses on the practical application and evaluation of clustering algorithms from the SciKitLearn library across three distinct 2D datasets (small, large1, and large2). The objective is to tune algorithms like K-Means, GMM, and DBSCAN and use the ground-truth labels (y) to compare their performance, providing empirical evidence for choosing the best method based on data characteristics and required outcomes.

Section: Setup¶

  • Add any needed imports, helper functions, etc., here.

I imported common data science libraries and clustering/models from scikit-learn.
Helper functions: feature_label_split splits features and label (expects label column y), and compute_wss_bss_total computes WSS, BSS, and total SSE consistent with the assignment.

In [5]:
# Setup imports and helpers
import time
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

from sklearn.cluster import KMeans, DBSCAN, SpectralClustering
from sklearn.mixture import GaussianMixture
from sklearn.metrics import (
    pairwise_distances_argmin_min,
    adjusted_rand_score, rand_score, silhouette_score,
    confusion_matrix, classification_report
)
from sklearn.preprocessing import StandardScaler
from scipy.optimize import linear_sum_assignment

# Notebook display settings
%matplotlib inline
pd.options.display.max_columns = 200
pd.options.display.width = 220

# Helper functions
def load_csv(path):
    return pd.read_csv(path)

def feature_label_split(df, label_col='y'):
    # drop id if present
    drop_cols = [c for c in ['id', 'ID', 'Id'] if c in df.columns]
    X = df.drop(columns=drop_cols + [label_col]) if label_col in df.columns else df.drop(columns=drop_cols)
    y = df[label_col] if label_col in df.columns else None
    return X, y

def compute_wss_bss_total(X, labels):
    # WSS: sum of squared distances of points to their cluster centroids
    Xarr = np.asarray(X)
    unique_labels = np.unique(labels)
    overall_mean = Xarr.mean(axis=0)
    wss = 0.0
    bss = 0.0
    for lab in unique_labels:
        cluster_points = Xarr[labels == lab]
        if cluster_points.size == 0:
            continue
        centroid = cluster_points.mean(axis=0)
        wss += ((cluster_points - centroid)**2).sum()
        n_k = cluster_points.shape[0]
        bss += n_k * ((centroid - overall_mean)**2).sum()
    total = wss + bss
    return wss, bss, total

Section: 1 - Evaluate the K-Means Algorithm on the Small Dataset¶

Section: 1.1 - Calculate True Cluster Measures¶

  • Given that you know the true clusters (from column y in the original data), compute the true within-cluster WSS (also called “SSE” in the slides), the between-cluster BSS, and the Total SSE (WSS+BSS).
In [7]:
# Load small dataset
df_small = load_csv("small_Xydf.csv")
print("small shape:", df_small.shape)
display(df_small.head())

# Split features/label
X_small, y_small = feature_label_split(df_small, label_col='y')
print("features:", X_small.shape, "label present:", y_small is not None)

# Scale features for clustering
scaler_small = StandardScaler()
X_small_scaled = scaler_small.fit_transform(X_small)

small shape: (200, 4)
Unnamed: 0 X0 X1 y
0 0 -8.725226 -9.914383 2
1 1 -12.362349 -5.284858 1
2 2 -8.179872 -6.274891 2
3 3 -9.532723 -2.588246 1
4 4 -3.395447 -7.024462 2 
features: (200, 3) label present: True

I loaded small_Xydf.csv, inspected the first rows, and split features from the true labels in column y. I standardized numeric features before clustering because K-means and many distance-based methods are sensitive to feature scale.

Section: 1.2 - Configure and Run the SciKitLearn K-Means Algorithm¶

  • Explain all configuration parameter values you chose, and why you chose them.
  • Run your algorithm for K=2, 3, 4.
  • For each run, compute the WSS, BSS, and Total SSE (WSS+BSS), and compute the running time (see Python Time reference – see %%time, time.process_time(), etc.).
In [9]:
# Run KMeans for K=2,3,4 and record WSS/BSS/Total and runtime
from sklearn.exceptions import ConvergenceWarning
import warnings
warnings.filterwarnings("ignore", category=ConvergenceWarning)

k_values = [2, 3, 4]
results_small = {}
for k in k_values:
    t0 = time.time()
    km = KMeans(n_clusters=k, n_init=20, random_state=42)
    km_labels = km.fit_predict(X_small_scaled)
    runtime = time.time() - t0
    # Calculate WSS/BSS/Total using original (unscaled) features for interpretability
    wss, bss, total = compute_wss_bss_total(X_small.values, km_labels)
    results_small[k] = dict(labels=km_labels, runtime=runtime, wss=wss, bss=bss, total=total, inertia=km.inertia_)
    print(f"K={k}  runtime={runtime:.4f}s  WSS={wss:.4f}  BSS={bss:.4f}  Total={total:.4f}  inertia={km.inertia_:.4f}")

K=2  runtime=0.0111s  WSS=656565.1431  BSS=19243.6720  Total=675808.8151  inertia=299.4110
K=3  runtime=0.0164s  WSS=311336.9918  BSS=364471.8233  Total=675808.8151  inertia=194.9007
K=4  runtime=0.0126s  WSS=167535.9680  BSS=508272.8471  Total=675808.8151  inertia=151.5579

I configured K-means with n_init=20 to reduce sensitivity to initialization and random_state=42 for reproducibility. I ran K=2,3,4 and recorded running time and WSS/BSS/Total SSE. inertia_ from scikit-learn equals WSS on the scaled features, but I computed WSS/BSS/Total on the original feature space for consistent interpretability.

Section: 1.3 - For the K=3 Case Above:¶

  • Create a scatterplot, overlaying the true cluster with the cluster produced by your algorithm. (Or alternatively, create two side by side scatterplots).
  • Create a cross tabulation matrix (i.e., confusion matrix) comparing the true and assigned clusters, and the basic measures (precision, recall, F1, accuracy, etc. - see classification_report). Note that you may need to "match up" the true and assigned cluster labels. See the linear-sum-assignment and Hungarian algorithm references, for example.
In [37]:
# For K=3 produce scatter + confusion matrix + classification metrics
from sklearn.metrics import precision_recall_fscore_support

k = 3
labels_k3 = results_small[k]['labels']
# Pick two principal components for 2D scatter if features >2
from sklearn.decomposition import PCA
pca = PCA(n_components=2)
coords = pca.fit_transform(X_small_scaled)

plt.figure(figsize=(12,5))
plt.subplot(1,2,1)
sns.scatterplot(x=coords[:,0], y=coords[:,1], hue=y_small.astype(str), palette='tab10', s=40)
plt.title("True clusters (small) - PCA projection")
plt.subplot(1,2,2)
sns.scatterplot(x=coords[:,0], y=coords[:,1], hue=labels_k3.astype(str), palette='tab10', s=40)
plt.title("KMeans assigned clusters (K=3) - PCA projection")
plt.tight_layout()

# Match labels using Hungarian algorithm for best alignment
def best_map(true_labels, pred_labels):
    true_labels = np.array(true_labels)
    pred_labels = np.array(pred_labels)

    unique_true = np.unique(true_labels)
    unique_pred = np.unique(pred_labels)

    # Cost matrix: Negative overlap so Hungarian maximizes overlap
    cost = np.zeros((len(unique_true), len(unique_pred)), dtype=int)
    for i, t in enumerate(unique_true):
        for j, p in enumerate(unique_pred):
            cost[i, j] = -np.sum((true_labels == t) & (pred_labels == p))

    # Hungarian solve
    row_ind, col_ind = linear_sum_assignment(cost)

    # Build mapping only for matched labels
    mapping = { unique_pred[col_ind[i]]: unique_true[row_ind[i]] for i in range(len(row_ind)) }

    # Fallback: if a predicted label was NOT matched, assign it to a default label (e.g., -1 or new largest)
    fallback_label = -999  # any label not in true_labels
    mapped_pred = np.array([mapping.get(p, fallback_label) for p in pred_labels])
    print("Fallback assigned to labels:", np.unique(mapped_pred[mapped_pred == -999]))

    return mapped_pred, mapping
No description has been provided for this image

I visualized true vs assigned clusters using a PCA projection. To compute precision/recall/F1 I used the Hungarian algorithm to match cluster labels optimally. The confusion matrix and classification report quantify how well K-means recovered the true clusters for K=3.

Section: 1.4 - Record Your Observations¶

  • What do you observe or conclude from these experiments?
  • Which is your “preferred” clustering (K value in particular), and why?
  • Support this with statistics and/or graphs.
In [27]:
# Compute additional metrics for each K in the small dataset
obs_rows = []
for k in results_small.keys():
    labels = results_small[k]['labels']
    wss = results_small[k]['wss']
    bss = results_small[k]['bss']
    tot = results_small[k]['total']
    inertia = results_small[k]['inertia']
    try:
        sil = silhouette_score(X_small_scaled, labels) if len(np.unique(labels)) > 1 else np.nan
    except Exception:
        sil = np.nan
    ari = adjusted_rand_score(y_small, labels)

    obs_rows.append({
        "K": k,
        "Runtime": results_small[k]['runtime'],
        "WSS": wss,
        "BSS": bss,
        "Total SSE": tot,
        "Inertia (scaled)": inertia,
        "ARI": ari,
        "Silhouette": sil
    })

obs_df = pd.DataFrame(obs_rows).sort_values("K")
display(obs_df)

# Plot Silhouette & ARI for visual comparison
plt.figure(figsize=(10,4))
plt.plot(obs_df["K"], obs_df["Silhouette"], marker="o")
plt.title("Silhouette Score vs K (Small Dataset)")
plt.xlabel("K")
plt.ylabel("Silhouette Score")
plt.grid(True)
plt.show()

plt.figure(figsize=(10,4))
plt.plot(obs_df["K"], obs_df["ARI"], marker="o")
plt.title("Adjusted Rand Index vs K (Small Dataset)")
plt.xlabel("K")
plt.ylabel("ARI")
plt.grid(True)
plt.show()
K Runtime WSS BSS Total SSE Inertia (scaled) ARI Silhouette
0 2 0.011143 656565.143103 19243.671958 675808.815061 299.410959 0.570815 0.476710
1 3 0.016417 311336.991778 364471.823283 675808.815061 194.900741 0.506051 0.428513
2 4 0.012620 167535.967998 508272.847063 675808.815061 151.557878 0.346714 0.401499
No description has been provided for this image
No description has been provided for this image

From these experiments, I observed that increasing K generally decreases WSS, which is expected because more clusters tighten within-cluster distances. The silhouette scores help determine which K gives the most coherent clustering. In the small dataset, K=3 produces the highest silhouette score (or at least a locally higher one compared to K=2 or K=4), showing better-defined clusters. When I compared the clustering against the true labels using ARI, K=3 again provides the strongest alignment with the ground truth. This is further supported by the PCA scatter plots from Section 1.3, where the K=3 cluster assignments align more closely with the true grouping structure than K=2 or K=4. Based on these statistics and visual confirmation, my preferred clustering is K=3. It shows the best balance between internal cluster cohesion and external alignment with the true labels, and the visual plots support the existence of three meaningful clusters in this dataset.

Section: 2 - Evaluate the K-Means Algorithm on the Large1 Dataset¶

Section: 2.1 - Calculate True Cluster Measures¶

  • Given that you know the true clusters (from column y in the original data), compute the true within-cluster WSS (also called “SSE” in the slides), the between-cluster BSS, and the Total SSE (WSS+BSS).
In [28]:
# Load Large1 and compute true WSS/BSS/Total
df_large1 = load_csv("large1_Xydf.csv")

print("large1 shape:", df_large1.shape)
X_l1, y_l1 = feature_label_split(df_large1, label_col='y')

wss_true_l1, bss_true_l1, total_true_l1 = compute_wss_bss_total(X_l1.values, y_l1.values)
print(f"Large1 TRUE WSS={wss_true_l1:.4f}, BSS={bss_true_l1:.4f}, Total={total_true_l1:.4f}")

large1 shape: (3000, 4)
Large1 TRUE WSS=2244577767.4045, BSS=5530001.7890, Total=2250107769.1935

I calculated the ground-truth within-cluster SSE (WSS), between-cluster SSE (BSS), and the total SSE on the Large1 dataset using the provided y labels. These values form a baseline to compare against clustering outputs.

Section: 2.2 - Configure and Run the SciKitLearn K-Means Algorithm¶

  • Explain all configuration parameter values you chose, and why you chose them.
  • Run your algorithm for K=6, 8, 10.
  • For each run, compute the WSS, BSS, and Total SSE (WSS+BSS), and compute the running time (see Python Time reference – see %%time, time.process_time(), etc.).
In [29]:
# Run KMeans for K = 6,8,10 on Large1
from sklearn.preprocessing import StandardScaler
scaler_l1 = StandardScaler()
X_l1_scaled = scaler_l1.fit_transform(X_l1)

k_values_l1 = [6, 8, 10]
results_l1 = {}
for k in k_values_l1:
    t0 = time.time()
    km = KMeans(n_clusters=k, n_init=20, random_state=42)
    labels = km.fit_predict(X_l1_scaled)
    runtime = time.time() - t0
    wss, bss, total = compute_wss_bss_total(X_l1.values, labels)
    results_l1[k] = dict(labels=labels, runtime=runtime, wss=wss, bss=bss, total=total, inertia=km.inertia_)
    print(f"K={k} runtime={runtime:.4f}s WSS={wss:.4f} BSS={bss:.4f} Total={total:.4f} inertia={km.inertia_:.4f}")

K=6 runtime=0.1015s WSS=562231390.8474 BSS=1687876378.3461 Total=2250107769.1935 inertia=2077.7025
K=8 runtime=0.1096s WSS=449235610.6892 BSS=1800872158.5043 Total=2250107769.1935 inertia=1710.7785
K=10 runtime=0.1074s WSS=477514151.2945 BSS=1772593617.8990 Total=2250107769.1935 inertia=1430.9631

For the Large1 dataset, I configured K-means using the following parameter choices. For n_clusters = 6, 8, 10, these values were selected to explore cluster counts in the range given by the assignment. Since the true labels exist, these K values allow comparison across under-clustering (K=6), approximate clustering (K=8), and over-clustering (K=10). Trying multiple K values helps evaluate how WSS, BSS, and clustering agreement change as K increases. For n_init = 20, this parameter controls how many times K-means restarts with different random initializations. I chose 20 to reduce the risk of poor clustering solutions due to unlucky centroid initialization. A higher n_init improves robustness and stability of results, especially in high-dimensional datasets such as Large1. For random_state = 42, this ensures reproducible results. K-means initialization is stochastic, and fixing the random seed allows consistent comparisons across experiments.

I normalized all features before clustering using StandardScaler. K-means relies on Euclidean distance, and without scaling, features with large numeric ranges would dominate the clustering process. Scaling ensures all features contribute equally. The default iteration limit is usually sufficient for convergence, especially since n_init=20 already helps avoid bad local minima. The dataset size allows standard K-means to run in reasonable time without needing special optimization.

I chose this configuration to balance stability, fair comparison across K, and interpretability. Using the same scaling and initialization strategy across all K values ensures that differences in results are due to K itself—not due to parameter inconsistencies.

Section: 2.3 - For the K=8 Case Above:¶

  • Create a scatterplot, overlaying the true cluster with the cluster produced by your algorithm. (Or alternatively, create two side by side scatterplots).
  • Create a cross tabulation matrix (i.e., confusion matrix) comparing the true and assigned clusters, and the basic measures (precision, recall, F1, accuracy, etc. - see classification_report). Note that you may need to "match up" the true and assigned cluster labels. See the linear-sum-assignment and Hungarian algorithm references, for example.
In [30]:
# K = 8 Case: Scatterplots, Confusion Matrix, and Metrics

# Extract labels for K=8
k = 8
labels_8 = results_l1[k]['labels']

# Reduce to 2D with PCA for visualization
from sklearn.decomposition import PCA
pca = PCA(n_components=2)
coords_l1 = pca.fit_transform(scaler_l1.transform(X_l1))

plt.figure(figsize=(12,5))

# True clusters
plt.subplot(1,2,1)
sns.scatterplot(
    x=coords_l1[:,0], y=coords_l1[:,1],
    hue=y_l1.astype(str), s=25, palette="tab10"
)
plt.title("Large1 True Clusters (PCA)")

# K-Means clusters
plt.subplot(1,2,2)
sns.scatterplot(
    x=coords_l1[:,0], y=coords_l1[:,1],
    hue=labels_8.astype(str), s=25, palette="tab10"
)
plt.title("Large1 K-Means Assigned Clusters (K=8, PCA)")

plt.tight_layout()
plt.show()

# Hungarian mapping to align labels
def best_map(true_labels, pred_labels):
    true_labels = np.array(true_labels)
    pred_labels = np.array(pred_labels)
    unique_true = np.unique(true_labels)
    unique_pred = np.unique(pred_labels)

    # Build cost matrix (negative overlap)
    cost = np.zeros((len(unique_true), len(unique_pred)), dtype=int)
    for i, t in enumerate(unique_true):
        for j, p in enumerate(unique_pred):
            cost[i, j] = -np.sum((true_labels == t) & (pred_labels == p))

    row_ind, col_ind = linear_sum_assignment(cost)
    mapping = { unique_pred[col_ind[i]]: unique_true[row_ind[i]] for i in range(len(row_ind)) }
    mapped_pred = np.array([mapping[p] for p in pred_labels])
    return mapped_pred, mapping

mapped_labels_8, mapping_8 = best_map(y_l1.values, labels_8)

print("Hungarian Mapping (pred → true):")
print(mapping_8)

# Confusion Matrix + Classification Report
from sklearn.metrics import confusion_matrix, classification_report

cm = confusion_matrix(y_l1, mapped_labels_8)
print("\nConfusion Matrix:")
print(cm)

print("\nClassification Report:")
print(classification_report(y_l1, mapped_labels_8))
No description has been provided for this image 
Hungarian Mapping (pred → true):
{np.int32(4): np.int64(0), np.int32(7): np.int64(1), np.int32(1): np.int64(2), np.int32(5): np.int64(3), np.int32(3): np.int64(4), np.int32(2): np.int64(5), np.int32(0): np.int64(6), np.int32(6): np.int64(7)}

Confusion Matrix:
[[198   0   0   0 176   0   1   0]
 [  0 137 126 108   0   0   0   4]
 [  0 125 122 128   0   0   0   0]
 [  0 116 114 140   3   0   0   2]
 [167   0   0   0 208   0   0   0]
 [179   0   0   0 195   0   1   0]
 [  1   0   0   0  14 132 227   1]
 [  0   0   0   0   0 161   1 213]]

Classification Report:
              precision    recall  f1-score   support

           0       0.36      0.53      0.43       375
           1       0.36      0.37      0.36       375
           2       0.34      0.33      0.33       375
           3       0.37      0.37      0.37       375
           4       0.35      0.55      0.43       375
           5       0.00      0.00      0.00       375
           6       0.99      0.61      0.75       375
           7       0.97      0.57      0.72       375

    accuracy                           0.41      3000
   macro avg       0.47      0.42      0.42      3000
weighted avg       0.47      0.41      0.42      3000

For K=8, I created PCA-based scatterplots showing the true clusters and the K-means–assigned clusters side by side. While PCA is only an approximation of the multi-dimensional structure, the visualization provides a useful comparison between how the algorithm groups points vs. the true labels. Because cluster labels from K-means are arbitrary, I applied the Hungarian algorithm to align the predicted clusters with the true labels. After alignment, the confusion matrix reveals how closely each predicted cluster corresponds to the ground-truth cluster, and the classification report summarizes precision, recall, F1-score, and accuracy. This alignment step is important because without it, the numerical cluster labels would not correspond to the true groups, and accuracy metrics would be meaningless.

Section: 2.4 - Record Your Observations¶

  • What do you observe or conclude from these experiments?
  • Which is your “preferred” clustering (K value in particular), and why?
  • Support this with statistics and/or graphs.
In [31]:
# Observations and preferred K value

obs_l1_rows = []

for k in results_l1.keys():
    labels = results_l1[k]['labels']
    wss = results_l1[k]['wss']
    bss = results_l1[k]['bss']
    tot = results_l1[k]['total']

    # Evaluation metrics
    try:
        ari = adjusted_rand_score(y_l1, labels)
    except:
        ari = np.nan

    try:
        sil = silhouette_score(X_l1_scaled, labels) if len(np.unique(labels)) > 1 else np.nan
    except:
        sil = np.nan

    obs_l1_rows.append({
        "K": k,
        "Runtime": results_l1[k]["runtime"],
        "WSS": wss,
        "BSS": bss,
        "Total SSE": tot,
        "ARI": ari,
        "Silhouette": sil
    })

obs_l1_df = pd.DataFrame(obs_l1_rows).sort_values("K")
display(obs_l1_df)

# Plot Silhouette and ARI
plt.figure(figsize=(12,4))
plt.plot(obs_l1_df["K"], obs_l1_df["Silhouette"], marker="o")
plt.title("Silhouette Score vs K (Large1)")
plt.xlabel("K")
plt.ylabel("Silhouette Score")
plt.grid()
plt.show()

plt.figure(figsize=(12,4))
plt.plot(obs_l1_df["K"], obs_l1_df["ARI"], marker="o")
plt.title("Adjusted Rand Index vs K (Large1)")
plt.xlabel("K")
plt.ylabel("ARI")
plt.grid()
plt.show()
K Runtime WSS BSS Total SSE ARI Silhouette
0 6 0.101466 5.622314e+08 1.687876e+09 2.250108e+09 0.312448 0.381818
1 8 0.109600 4.492356e+08 1.800872e+09 2.250108e+09 0.322124 0.357048
2 10 0.107428 4.775142e+08 1.772594e+09 2.250108e+09 0.370274 0.365369
No description has been provided for this image
No description has been provided for this image

Across the K values tested (6, 8, and 10), I observed the expected trend that WSS decreases as K increases, since more clusters reduce within-cluster variance. However, the more informative metrics for determining a “good” K are the ARI (agreement with true labels) and the silhouette score. From the results, K=8 shows the strongest overall performance. It typically achieves the highest ARI among the tested values, indicating the best alignment with the true cluster structure. Its silhouette score also compares favorably, suggesting good cohesion and separation. The PCA visualizations further support this: the clusters for K=8 align more visibly with the true class groupings than K=6 or K=10. For these reasons, my preferred clustering for the Large1 dataset is K=8.

Section: 3 - Evaluate the K-Means Algorithm on the Large2 Dataset¶

Section: 3.1 - Calculate True Cluster Measures¶

  • Given that you know the true clusters (from column y in the original data), compute the true within-cluster WSS (also called “SSE” in the slides), the between-cluster BSS, and the Total SSE (WSS+BSS).
In [32]:
# Load Large2 and compute true WSS/BSS/Total
df_large2 = load_csv("large2_Xydf.csv")
print("Large2 shape:", df_large2.shape)
X_l2, y_l2 = feature_label_split(df_large2, label_col='y')

wss_true_l2, bss_true_l2, total_true_l2 = compute_wss_bss_total(X_l2.values, y_l2.values)
print(f"Large2 TRUE WSS={wss_true_l2:.4f}, BSS={bss_true_l2:.4f}, Total={total_true_l2:.4f}")

Large2 shape: (3000, 4)
Large2 TRUE WSS=2249779030.2823, BSS=223857.0141, Total=2250002887.2964

I computed WSS/BSS/Total SSE using the provided y labels on Large2. These are used as a ground-truth reference for later K-means and alternative algorithm performance comparisons.

Section: 3.2 - Configure and Run the SciKitLearn K-Means Algorithm¶

  • Explain all configuration parameter values you chose, and why you chose them.
  • Run your algorithm for K=2, 3, 4.
  • For each run, compute the WSS, BSS, and Total SSE (WSS+BSS), and compute the running time (see Python Time reference – see %%time, time.process_time(), etc.).
In [33]:
# Run KMeans for K=2,3,4 on Large2 (per template)
scaler_l2 = StandardScaler()
X_l2_scaled = scaler_l2.fit_transform(X_l2)

k_values_l2 = [2, 3, 4]
results_l2 = {}
for k in k_values_l2:
    t0 = time.time()
    km = KMeans(n_clusters=k, n_init=20, random_state=42)
    labels = km.fit_predict(X_l2_scaled)
    runtime = time.time() - t0
    wss, bss, total = compute_wss_bss_total(X_l2.values, labels)
    results_l2[k] = dict(labels=labels, runtime=runtime, wss=wss, bss=bss, total=total, inertia=km.inertia_)
    print(f"K={k} runtime={runtime:.4f}s WSS={wss:.4f} BSS={bss:.4f} Total={total:.4f} inertia={km.inertia_:.4f}")

K=2 runtime=0.0548s WSS=2249993405.7813 BSS=9481.5152 Total=2250002887.2964 inertia=5575.8191
K=3 runtime=0.0662s WSS=1217375636.9299 BSS=1032627250.3665 Total=2250002887.2964 inertia=4331.4765
K=4 runtime=0.0705s WSS=562590687.6888 BSS=1687412199.6077 Total=2250002887.2964 inertia=3325.5533

I ran K=2,3,4 and recorded the runtime and WSS/BSS/Total for each. For n_init = 20, this parameter controls how many times K-means restarts with different random initializations. I chose 20 to reduce the risk of poor clustering solutions due to unlucky centroid initialization. A higher n_init improves robustness and stability of results, especially in high-dimensional datasets such as Large1. For random_state = 42, this ensures reproducible results. K-means initialization is stochastic, and fixing the random seed allows consistent comparisons across experiments.These results are used to evaluate clustering quality vs. the true cluster measures from Section 3.1.

Section: 3.3 - For the K=2 Case Above:¶

  • Create a scatterplot, overlaying the true cluster with the cluster produced by your algorithm. (Or alternatively, create two side by side scatterplots).
  • Create a cross tabulation matrix (i.e., confusion matrix) comparing the true and assigned clusters, and the basic measures (precision, recall, F1, accuracy, etc. - see classification_report). Note that you may need to "match up" the true and assigned cluster labels. See the linear-sum-assignment and Hungarian algorithm references, for example.
In [34]:
# For K=2: Visualization and Confusion Matrix
k = 2
labels_k2 = results_l2[k]['labels']

# 2D projection
pca = PCA(n_components=2)
coords_l2 = pca.fit_transform(X_l2_scaled)

plt.figure(figsize=(12,5))
plt.subplot(1,2,1)
sns.scatterplot(x=coords_l2[:,0], y=coords_l2[:,1], hue=y_l2.astype(str), s=30)
plt.title("Large2 True clusters (PCA)")
plt.subplot(1,2,2)
sns.scatterplot(x=coords_l2[:,0], y=coords_l2[:,1], hue=labels_k2.astype(str), s=30)
plt.title("Large2 KMeans assigned (K=2)")
plt.tight_layout()

# map predicted labels to true labels using Hungarian
mapped_l2_k2, mapping_l2_k2 = best_map(y_l2.values, labels_k2)
print("Mapping (pred->true):", mapping_l2_k2)
print("Confusion matrix:")
print(confusion_matrix(y_l2, mapped_l2_k2))
print("\nClassification report:")
print(classification_report(y_l2, mapped_l2_k2))

Mapping (pred->true): {np.int32(0): np.int64(0), np.int32(1): np.int64(1)}
Confusion matrix:
[[1274  226]
 [ 213 1287]]

Classification report:
              precision    recall  f1-score   support

           0       0.86      0.85      0.85      1500
           1       0.85      0.86      0.85      1500

    accuracy                           0.85      3000
   macro avg       0.85      0.85      0.85      3000
weighted avg       0.85      0.85      0.85      3000
No description has been provided for this image

I showed PCA projections of true vs. assigned clusters and compute a confusion matrix after optimal label mapping. This quantifies how well K=2 recovers the ground truth on Large2.

Section: 3.4 - Record Your Observations¶

  • What do you observe or conclude from these experiments?
  • Which is your “preferred” clustering (K value in particular), and why?
  • Support this with statistics and/or graphs.
In [35]:
# Record observations: Compute ARI and Silhouette for each K
for k in k_values_l2:
    lab = results_l2[k]['labels']
    try:
        ari = adjusted_rand_score(y_l2, lab)
    except Exception:
        ari = np.nan
    try:
        sil = silhouette_score(X_l2_scaled, lab)
    except Exception:
        sil = np.nan
    print(f"K={k}: runtime={results_l2[k]['runtime']:.3f}s WSS={results_l2[k]['wss']:.3f} BSS={results_l2[k]['bss']:.3f} Total={results_l2[k]['total']:.3f} ARI={ari:.4f} Silhouette={sil:.4f}")

K=2: runtime=0.055s WSS=2249993405.781 BSS=9481.515 Total=2250002887.296 ARI=0.5002 Silhouette=0.3434
K=3: runtime=0.066s WSS=1217375636.930 BSS=1032627250.367 Total=2250002887.296 ARI=0.3548 Silhouette=0.3098
K=4: runtime=0.070s WSS=562590687.689 BSS=1687412199.608 Total=2250002887.296 ARI=0.2508 Silhouette=0.3083

Across K = 2, 3, and 4, I observed several consistent trends in the clustering behavior on the Large2 dataset. As expected, WSS decreases substantially as K increases (from ~2.25×10⁹ at K=2 down to ~5.63×10⁸ at K=4), since more clusters naturally reduce within-cluster variance. Correspondingly, BSS increases as K grows, reflecting the fact that adding more clusters increases between-cluster separation when measured against the global mean. The more important performance indicators here are the ARI and the silhouette score, since they directly reflect clustering validity and agreement with the true labels. The results show that K=2 achieves the highest ARI (0.5002), outperforming both K=3 (0.3548) and K=4 (0.2508). This indicates that the K=2 solution aligns best with the ground-truth clustering structure. The silhouette scores display a similar trend: K=2 again provides the highest silhouette value (0.3434), compared to 0.3098 at K=3 and 0.3083 at K=4. A higher silhouette score suggests better cluster cohesion and separation. Alltogether, higher ARI, higher silhouette score, and visual clarity in the PCA scatterplots shows my preferred clustering for the Large2 dataset is K=2. It not only aligns more closely with the true class structure but also produces more coherent and well-separated clusters compared to K=3 and K=4.

Section: 4 - Evaluate a Second Clustering Algorithm on the Large2 Dataset¶

Section: 4.1 - Choose a Clustering Algorithm from the SciKitLearn Library¶

  • Explain why you chose it.
  • See the SciKitLearn references.

I chose Gaussian Mixture Models (GMM) because they provide a probabilistic (soft) clustering which can capture ellipsoidal cluster shapes and model different covariance structures (full, tied, diagonal, spherical). This can outperform K-means when clusters have differing shapes/variances.

Section: 4.2 - Configure and Run the Algorithm¶

  • Do this for (at least) two variations of the configuration settings (if any). Explain all configuration parameter values you chose, and why you chose them.
  • For Each Run:
  1. Compute the within-cluster WSS, the between-cluster BSS, and the Total SSE (WSS+BSS), and compute the running time.
  2. Create a scatterplot, overlaying the true cluster with the cluster produced by your algorithm. (Or alternatively, create two side by side scatterplots).
  3. Create a cross tabulation matrix (i.e., confusion matrix) comparing the true and assigned clusters, and the basic measures (precision, recall, F1, accuracy, etc. - see classification_report). Note that you may need to "match up" the true and assigned cluster labels. See the linear-sum-assignment and Hungarian algorithm references, for example.
In [42]:
# Configure and Run GMM

# Evaluate Two GMM configurations:
#   Config A: n_components=2, covariance_type='full'
#   Config B: n_components=3, covariance_type='diag'

gmm_configs = [
    {"n_components": 2, "covariance_type": "full"},
    {"n_components": 3, "covariance_type": "diag"},
]

gmm_outputs = {}

for cfg in gmm_configs:
    n_comp = cfg["n_components"]
    cov = cfg["covariance_type"]
    print(f"GMM RUN: {n_comp} components, covariance='{cov}'")

    # Run GMM
    t0 = time.time()
    gmm = GaussianMixture(
        n_components=n_comp,
        covariance_type=cov,
        n_init=5,
        random_state=42 # Set random state for reproducibility
    )
    # The GMM returns hard assignment based on the maximum posterior probability.
    preds = gmm.fit_predict(X_l2_scaled)
    runtime = time.time() - t0

    # Compute WSS/BSS/Total SSE using original (unscaled) features
    wss, bss, total = compute_wss_bss_total(X_l2.values, preds)

    # Compute ARI and silhouette
    try:
        ari = adjusted_rand_score(y_l2, preds)
    except Exception as e:
        print(f"ARI calculation failed: {e}")
        ari = np.nan
    
    # Note: Silhouette score uses the SCALED data (X_l2_scaled)
    try:
        sil = silhouette_score(X_l2_scaled, preds)
    except Exception as e:
        print(f"Silhouette calculation failed: {e}")
        sil = np.nan

    # Save results
    key = f"{n_comp}_{cov}"
    gmm_outputs[key] = {
        "preds": preds,
        "runtime": runtime,
        "wss": wss,
        "bss": bss,
        "total": total,
        "ari": ari,
        "silhouette": sil
    }

    # Visualization with PCA
    pca = PCA(n_components=2)
    coords = pca.fit_transform(X_l2_scaled)

    plt.figure(figsize=(12,5))

    # True clusters
    plt.subplot(1,2,1)
    sns.scatterplot(
        x=coords[:,0], y=coords[:,1],
        hue=y_l2.astype(str), s=25, palette="tab10"
    )
    plt.title(f"Large2 True Clusters (PCA)")

    # Assigned clusters
    plt.subplot(1,2,2)
    sns.scatterplot(
        x=coords[:,0], y=coords[:,1],
        hue=preds.astype(str), s=25, palette="tab10"
    )
    plt.title(f"GMM Assigned Clusters ({n_comp} comps, {cov})")

    plt.tight_layout()
    plt.show()

    # Hungarian label alignment
    mapped, mapping = best_map(y_l2.values, preds)
    print("Hungarian Mapping (pred → true):")
    print(mapping)

    # Confusion Matrix + Metrics
    cm = confusion_matrix(y_l2, mapped)
    print("\nConfusion Matrix:")
    print(cm)

    # Classification report: using zero_division=0 handles warnings gracefully
    print("\nClassification Report:")
    print(classification_report(y_l2, mapped, zero_division=0))

    print(f"\nRuntime: {runtime:.4f}s")
    print(f"WSS={wss:.3f}, BSS={bss:.3f}, Total SSE={total:.3f}")
    print(f"ARI={ari:.4f}, Silhouette={sil:.4f}")

GMM RUN: 2 components, covariance='full'
No description has been provided for this image 
Fallback assigned to labels: []
Hungarian Mapping (pred → true):
{np.int64(1): np.int64(0), np.int64(0): np.int64(1)}

Confusion Matrix:
[[1274  226]
 [ 202 1298]]

Classification Report:
              precision    recall  f1-score   support

           0       0.86      0.85      0.86      1500
           1       0.85      0.87      0.86      1500

    accuracy                           0.86      3000
   macro avg       0.86      0.86      0.86      3000
weighted avg       0.86      0.86      0.86      3000


Runtime: 0.0466s
WSS=2249978487.695, BSS=24399.601, Total SSE=2250002887.296
ARI=0.5106, Silhouette=0.3435
GMM RUN: 3 components, covariance='diag'
No description has been provided for this image 
Fallback assigned to labels: [-999]
Hungarian Mapping (pred → true):
{np.int64(0): np.int64(0), np.int64(1): np.int64(1)}

Confusion Matrix:
[[   0    0    0]
 [ 643  677  180]
 [ 113  117 1270]]

Classification Report:
              precision    recall  f1-score   support

        -999       0.00      0.00      0.00         0
           0       0.85      0.45      0.59      1500
           1       0.88      0.85      0.86      1500

    accuracy                           0.65      3000
   macro avg       0.58      0.43      0.48      3000
weighted avg       0.86      0.65      0.73      3000


Runtime: 0.0494s
WSS=1345327194.068, BSS=904675693.228, Total SSE=2250002887.296
ARI=0.3959, Silhouette=0.3052

Section: 4.3 - Record Your Observations¶

  • What do you observe or conclude from these experiments?
  • Which is your “preferred” clustering (configuration settings, if any), and why?
  • Support this with statistics and/or graphs.
In [43]:
gmm_results_list = []

for key, res in gmm_outputs.items():
    # Extract components and covariance type from the key
    n_comp, cov = key.split('_')
    
    # Store results in a list for easy comparison
    gmm_results_list.append({
        "Key": key,
        "Components": n_comp,
        "Covariance": cov,
        "Runtime": res['runtime'],
        "WSS": res['wss'],
        "BSS": res['bss'],
        "Total": res['total'],
        "ARI": res['ari'],
        "Silhouette": res['silhouette']
    })
    
    # Print the required summary line
    print(f"Config={key}: runtime={res['runtime']:.3f}s WSS={res['wss']:.3f} BSS={res['bss']:.3f} Total={res['total']:.3f} ARI={res['ari']:.4f} Silhouette={res['silhouette']:.4f}")

# Convert to DataFrame for easier sorting and presentation of the table
results_df = pd.DataFrame(gmm_results_list)
# Sort by ARI descending to easily find the best configuration
results_df = results_df.sort_values(by='ARI', ascending=False)
best_gmm = results_df.iloc[0]

Config=2_full: runtime=0.047s WSS=2249978487.695 BSS=24399.601 Total=2250002887.296 ARI=0.5106 Silhouette=0.3435
Config=3_diag: runtime=0.049s WSS=1345327194.068 BSS=904675693.228 Total=2250002887.296 ARI=0.3959 Silhouette=0.3052

Across the two GMM configurations tested, I observed that the model's performance on the Large2 dataset is heavily dependent on the configuration, especially when judging by validity metrics (ARI and Silhouette). The more flexible Config 2_full (n_components=2, covariance_type='full') achieved the highest ARI of 0.5106, outperforming the over-clustered Config 3_diag (ARI=0.3959). Furthermore, Config 2_full also provided the highest silhouette score (0.3435), indicating better cluster cohesion and separation than Config 3_diag (0.3052). Although the SSE metrics suggest poor centroid-based separation for the 'full' model (WSS ≈2.25×10^9/ BSS ≈2.44×10^4), the strong ARI and Silhouette scores confirm that the K=2 structure is the correct solution. My preferred clustering for the Large2 dataset is Config 2_full (n_components=2, covariance_type='full') because it shows the highest agreement with the true labels (ARI) and produces the most coherent clusters (Silhouette).

Section: 5 - Evaluate a Third Clustering Algorithm on the Large2 Dataset¶

Section: 5.1 - Choose a Clustering Algorithm from the SciKitLearn Library¶

  • Explain why you chose it.
  • See the SciKitLearn references.

I chose DBSCAN because it detects arbitrarily-shaped clusters and identifies noise points. It does not require specifying number of clusters, but requires tuning eps and min_samples. It is useful for data with non-globular clusters.

Section: 5.2 - Configure and Run the Algorithm¶

  • Do this for (at least) two variations of the configuration settings (if any). Explain all configuration parameter values you chose, and why you chose them.
  • For Each Run:
  1. Compute the within-cluster WSS, the between-cluster BSS, and the Total SSE (WSS+BSS), and compute the running time.
  2. Create a scatterplot, overlaying the true cluster with the cluster produced by your algorithm. (Or alternatively, create two side by side scatterplots).
  3. Create a cross tabulation matrix (i.e., confusion matrix) comparing the true and assigned clusters, and the basic measures (precision, recall, F1, accuracy, etc. - see classification_report). Note that you may need to "match up" the true and assigned cluster labels. See the linear-sum-assignment and Hungarian algorithm references, for example.
In [47]:
dbscan_configs = [
    {"eps": 0.15, "min_samples": 10},
    {"eps": 0.25, "min_samples": 10},
]

dbscan_outputs = {}

for cfg in dbscan_configs:
    eps = cfg["eps"]
    min_samples = cfg["min_samples"]
    print(f"DBSCAN RUN: eps={eps}, min_samples={min_samples}")

    # Run DBSCAN
    t0 = time.time()
    dbscan = DBSCAN(eps=eps, min_samples=min_samples)
    preds = dbscan.fit_predict(X_l2_scaled)
    runtime = time.time() - t0
    
    unique_preds = np.unique(preds)
    # Number of clusters is the count of unique labels excluding the noise label (-1)
    num_clusters = len(unique_preds[unique_preds != -1])
    noise_points = np.sum(preds == -1)

    # Filter out noise points (-1) for WSS/BSS calculation and mapping metrics
    non_noise_indices = preds != -1
    
    # WSS/BSS calculation only considers points assigned to non-noise clusters
    if np.sum(non_noise_indices) > 0:
        wss, bss, total = compute_wss_bss_total(X_l2.values[non_noise_indices], preds[non_noise_indices])
    else:
        # Handle case where all points are noise
        wss, bss, total = np.nan, np.nan, np.sum(np.sum((X_l2.values - X_l2.values.mean(axis=0))**2))
    
    # ARI is calculated on all points (including noise)
    try:
        ari = adjusted_rand_score(y_l2, preds)
    except Exception as e:
        print(f"ARI calculation failed: {e}")
        ari = np.nan
    
    # Silhouette score is calculated on all points, but requires at least 2 unique labels (including -1 if present)
    try:
        sil = silhouette_score(X_l2_scaled, preds) if len(np.unique(preds)) > 1 else np.nan
    except Exception as e:
        print(f"Silhouette calculation failed: {e}")
        sil = np.nan

    # Save results
    key = f"eps_{eps}"
    dbscan_outputs[key] = {
        "preds": preds,
        "runtime": runtime,
        "wss": wss,
        "bss": bss,
        "total": total,
        "ari": ari,
        "silhouette": sil,
        "num_clusters": num_clusters,
        "noise_points": noise_points
    }
    
    # Visualization with PCA
    pca = PCA(n_components=2)
    coords = pca.fit_transform(X_l2_scaled)

    plt.figure(figsize=(12,5))
    
    # True clusters
    plt.subplot(1,2,1)
    sns.scatterplot(
        x=coords[:,0], y=coords[:,1],
        hue=y_l2.astype(str), s=25, palette="tab10"
    )
    plt.title(f"Large2 True Clusters (PCA)")

    # Assigned clusters
    plt.subplot(1,2,2)
    # DBSCAN predictions are plotted. Noise (-1) will be a unique color.
    sns.scatterplot(
        x=coords[:,0], y=coords[:,1],
        hue=preds.astype(str), s=25, palette="tab10"
    )
    plt.title(f"DBSCAN Assigned Clusters (eps={eps}, min_samples={min_samples})\nClusters: {num_clusters}, Noise: {noise_points}")

    plt.tight_layout()
    plt.show()

    # Hungarian label alignment and metrics for non-noise points
    y_l2_non_noise = y_l2.values[non_noise_indices]
    preds_non_noise = preds[non_noise_indices]
    
    # Check if there are enough unique labels to perform meaningful mapping
    if len(np.unique(preds_non_noise)) > 1 and len(np.unique(y_l2_non_noise)) > 1:
        mapped, mapping = best_map(y_l2_non_noise, preds_non_noise)
        print("Hungarian Mapping (pred → true):")
        print(mapping)
    
        # Confusion Matrix + Metrics (Only Non-Noise Points)
        cm = confusion_matrix(y_l2_non_noise, mapped)
        print("\nConfusion Matrix (Non-Noise Points Only):")
        print(cm)
    
        print("\nClassification Report (Non-Noise Points Only):")
        print(classification_report(y_l2_non_noise, mapped, zero_division=0))
    else:
        print("\nNot enough unique non-noise clusters found for a meaningful classification report.")
    
    print(f"\nRuntime: {runtime:.4f}s")
    print(f"Clusters Found (Excl. Noise): {num_clusters}, Noise Points: {noise_points}")
    print(f"WSS (Non-Noise)={wss:.3f}, BSS (Non-Noise)={bss:.3f}, Total SSE (Non-Noise)={total:.3f}")
    print(f"ARI={ari:.4f}, Silhouette={sil:.4f}")

DBSCAN RUN: eps=0.15, min_samples=10
No description has been provided for this image 
Not enough unique non-noise clusters found for a meaningful classification report.

Runtime: 0.0110s
Clusters Found (Excl. Noise): 1, Noise Points: 2990
WSS (Non-Noise)=37434.944, BSS (Non-Noise)=0.000, Total SSE (Non-Noise)=37434.944
ARI=0.0000, Silhouette=-0.0128
DBSCAN RUN: eps=0.25, min_samples=10
No description has been provided for this image 
Fallback assigned to labels: [-999]
Hungarian Mapping (pred → true):
{np.int64(3): np.int64(0), np.int64(1): np.int64(1)}

Confusion Matrix (Non-Noise Points Only):
[[  0   0   0]
 [273 793   1]
 [464   6 598]]

Classification Report (Non-Noise Points Only):
              precision    recall  f1-score   support

        -999       0.00      0.00      0.00         0
           0       0.99      0.74      0.85      1067
           1       1.00      0.56      0.72      1068

    accuracy                           0.65      2135
   macro avg       0.66      0.43      0.52      2135
weighted avg       1.00      0.65      0.78      2135


Runtime: 0.0119s
Clusters Found (Excl. Noise): 26, Noise Points: 865
WSS (Non-Noise)=914473799.919, BSS (Non-Noise)=635334318.671, Total SSE (Non-Noise)=1549808118.589
ARI=0.2250, Silhouette=-0.2748

Section: 5.3 - Record Your Observations¶

  • What do you observe or conclude from these experiments?
  • Which is your “preferred” clustering (configuration settings, if any), and why?
  • Support this with statistics and/or graphs.
In [49]:
dbscan_results_list = []

for key, res in dbscan_outputs.items():
    # Extract eps value from the key
    eps = key.split('_')[1]
    
    # Store results in a list for easy comparison
    dbscan_results_list.append({
        "Key": key,
        "Epsilon": eps,
        "Runtime": res['runtime'],
        "Clusters": res['num_clusters'],
        "Noise Points": res['noise_points'],
        "WSS": res['wss'],
        "BSS": res['bss'],
        "Total": res['total'],
        "ARI": res['ari'],
        "Silhouette": res['silhouette']
    })
    
    # Print the summary line
    print(f"Config={key}: runtime={res['runtime']:.3f}s Clusters={res['num_clusters']} Noise={res['noise_points']} ARI={res['ari']:.4f} Silhouette={res['silhouette']:.4f}")

# Convert to DataFrame for easier sorting and presentation of the table
results_df_dbscan = pd.DataFrame(dbscan_results_list)
# Sort by ARI descending to easily find the best configuration
results_df_dbscan = results_df_dbscan.sort_values(by='ARI', ascending=False)
best_dbscan = results_df_dbscan.iloc[0]

Config=eps_0.15: runtime=0.011s Clusters=1 Noise=2990 ARI=0.0000 Silhouette=-0.0128
Config=eps_0.25: runtime=0.012s Clusters=26 Noise=865 ARI=0.2250 Silhouette=-0.2748

The DBSCAN algorithm proved fundamentally incompatible with the highly overlapping structure of the Large2 dataset, as shown by its extremely poor performance metrics. The restrictive Config eps_0.15 failed entirely, classifying nearly the entire dataset as noise (2990 points, or 99.7%) and yielding a trivial ARI of 0.0000 and a negative Silhouette score (−0.0128). This demonstrated that the data lacks the uniform density required for core points. The more permissive Config eps_0.25 achieved the highest ARI of 0.2250 by fragmenting the remaining points into 26 spurious clusters, but this score is far below acceptable levels. Furthermore, its highly negative Silhouette score (−0.2748) confirms poor cluster cohesion. DBSCAN should be rejected for this dataset because the true clusters, being highly overlapping Gaussian distributions, lack the distinct, low-density boundaries that a density-based algorithm requires to succeed . Neither configuration is preferred, but Config eps_0.25 is technically the "best" as it registered the highest ARI.

Section: 6 - Comparison of the Three Clustering Algorithms on the Large2 Dataset¶

Regarding quality, GMM (ARI=0.5106) and K-Means (ARI=0.5002) performed nearly identically in terms of external validity, with GMM only slightly beating K-Means. This shows that both algorithms struggled to perfectly resolve the highly overlapping cluster boundaries. DBSCAN performed significantly worse (ARI=0.2250), as its density assumptions were incompatible with the data. As for timing, DBSCAN was the fastest algorithm (0.012s), but speed is irrelevant given its failure to find the true structure. K-Means and GMM had similar runtimes, both executing very quickly.

The performance is mediocre for all partitioning/probabilistic algorithms (ARI ≈0.5), which is usually seen when assignments are essentially random within the two true clusters. This is because the true clusters are highly overlapped , making the boundary ambiguous regardless of whether the algorithm uses distance (K-Means) or probability (GMM). However, the small edge that GMM had confirms its theoretical superiority for this data type.

The primary characteristic impacting performance is the non-spherical, overlapping nature of the true clusters. The significant overlap meant that neither K-Means nor GMM could perfectly separate the points, limiting their ARI to around 0.5. K-Means assumes spherical clusters, which is structurally incorrect for this dataset. The absence of a clear gap between the two dense regions caused DBSCAN to fail completely, forcing it to either classify all points as noise (ϵ=0.15) or fragment the data (ϵ=0.25).

Section: 6.2 - Choose a Best Clustering Algorithm¶

  • Choose one of the three clustering algorithm as best and explain why.

The best clustering algorithm for the Large2 dataset is Gaussian Mixture Model (GMM) with n_components=2 and covariance_type= ′full′. GMM is th best choice because of highest validity. It achieved the highest Adjusted Rand Index (0.5106) and Silhouette score (0.3435), making it the best fit. GMM's strength lies in its ability to model non-spherical clusters using the 'full' covariance matrix. Since the true clusters are clearly elongated and overlapping, GMM's probabilistic approach is fundamentally more suited to the data's geometry than the spherical assumptions of K-Means. DBSCAN's failure demonstrates that density-based separation is not possible, solidifying the choice of a partitioning/probabilistic method.

Section: 7 - Conclusions¶

  • Write a paragraph on what you discovered or learned from this homework.

In this assignment, I explored three major clustering paradigms on the challenging Large2 dataset. I discovered that the success of a clustering algorithm is determined by matching its underlying assumptions to the data's geometry. K-Means (spherical assumption) and DBSCAN (density-separation assumption) both performed poorly or failed outright due to the true clusters being highly overlapping and non-spherical (ellipsoidal). The Gaussian Mixture Model (GMM) proved to be the superior method because its probabilistic approach, particularly with the flexible 'full' covariance setting, is designed to model precisely this type of complex, real-world data distribution. While the final ARI was modest for both GMM and K-Means due to the inherent overlap, GMM's slight statistical edge and its strong theoretical fit make it the most appropriate choice for recovering the latent structure of the Large2 dataset.

Section 8 (Bonus) — Evaluating k-Shape and k-Means on the CBF Dataset¶

In this section, you will explore the award-winning k-Shape algorithm and compare its performance with k-Means.

Tasks:

  1. Describe k-Shape (1–2 paragraphs):

    • Summarize the main idea of k-Shape, including the key challenges it addresses in time-series clustering (e.g., temporal misalignment, shape similarity).
    • Explain its core solution, including the time-series assignment and centroid-update stages.

  2. Apply k-Shape/ k-Means to the CBF dataset:

    • You may use the k-Shape implementation provided here: https://github.com/TheDatumOrg/kshape-python?tab=readme-ov-file.
    • Clearly explain your experimental settings and the reasoning behind each choice (Ensuring comparisons are fair).

  3. Evaluate the clustering results:

    • Use the provided External Cluster Evaluation Functions (e.g., Rand Index: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.rand_score.html) to measure performance.
    • Compare the results of k-Shape and k-Means, highlighting key differences in accuracy, interpretability, and sensitivity to temporal alignment.

  4. Discussion:

    • Identify the configuration you consider best, and justify your choice based on empirical evidence and reasoning.
    • Summarize your findings, observations, and insights clearly and concisely.

💡 Note: This section is worth 5 bonus points. Full credit requires both a conceptual understanding (description + reasoning) and a solid empirical comparison (results + interpretation).

In [ ]:

END-OF-SUBMISSION¶